Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

_{Sortformer High-Latency • ES2004a • Runtime: 6m 10s • 2026-06-28T21:19:12.613Z}

Supertonic3 Smoke Test ✅

_{Runtime: 0m44s}

_{Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf.}

PocketTTS Smoke Test ✅

_{Runtime: 0m5s}

_{Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon.}

Parakeet EOU Benchmark Results ✅

Status: Benchmark passed
Chunk Size: 320ms
Files Tested: 100/100

Performance Metrics

Streaming Metrics

_{Test runtime: 1m22s • 06/28/2026, 04:58 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O}

VAD Benchmark Results

Performance Comparison

Dataset Details

• MUSAN: Music, Speech, and Noise dataset - standard VAD evaluation
• VOiCES: Voices Obscured in Complex Environmental Settings - tests robustness in real-world conditions

✅: Average F1-Score above 70%

ASR Benchmark Results ✅

Status: All benchmarks passed

Parakeet v3 (multilingual)

Parakeet v2 (English-optimized)

Streaming (v3)

Streaming (v2)

_{Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming}

_{25 files per dataset • Test runtime: 12m0s • 06/28/2026, 05:08 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)}

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

_{Testing methodology follows HuggingFace Open ASR Leaderboard}

✅ Nemotron Multilingual Benchmark — FLEURS

FLEURS en_us, chunk 2240ms, 100 samples, B1 fused decode path. Same English audio against both shipped models.

[latin / English]

Language     | Prompt   | WER%   | CER%   | RTFx   | Duration  | Processed | Skipped
--------------------------------------------------------------------------------
en_us        | en-US    | 7.8    | 3.5    | 6.3    | 953.9s    | 100       | -
--------------------------------------------------------------------------------
AVERAGE      | —        | 7.8    | 3.5    | 6.3   


[multilingual / English]

Language     | Prompt   | WER%   | CER%   | RTFx   | Duration  | Processed | Skipped
--------------------------------------------------------------------------------
en_us        | en-US    | 7.9    | 3.6    | 6.2    | 953.9s    | 100       | -
--------------------------------------------------------------------------------
AVERAGE      | —        | 7.9    | 3.6    | 6.2   

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Optimal clustering with Hungarian algorithm for maximum accuracy

Offline VBx Pipeline Timing Breakdown

Time spent in each stage of batch diarization

Speaker Diarization Research Comparison

Offline VBx achieves competitive accuracy with batch processing

Pipeline Details:

• Mode: Offline VBx with Hungarian algorithm for optimal speaker-to-cluster assignment
• Segmentation: VAD-based voice activity detection
• Embeddings: WeSpeaker-compatible speaker embeddings
• Clustering: PowerSet with VBx refinement
• Accuracy: Higher than streaming due to optimal post-hoc mapping

_{🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 134.4s processing • Test runtime: 2m 20s • 06/28/2026, 05:15 PM EST}

✅ Nemotron Multilingual Benchmark — FLEURS

FLEURS en_us, chunk 2240ms, 100 samples, B1 fused decode path. Same English audio against both shipped models.

[latin / English]

Language     | Prompt   | WER%   | CER%   | RTFx   | Duration  | Processed | Skipped
--------------------------------------------------------------------------------
en_us        | en-US    | 7.8    | 3.5    | 4.6    | 953.9s    | 100       | -
--------------------------------------------------------------------------------
AVERAGE      | —        | 7.8    | 3.5    | 4.6   


[multilingual / English]

Language     | Prompt   | WER%   | CER%   | RTFx   | Duration  | Processed | Skipped
--------------------------------------------------------------------------------
en_us        | en-US    | 7.9    | 3.6    | 4.7    | 953.9s    | 100       | -
--------------------------------------------------------------------------------
AVERAGE      | —        | 7.9    | 3.6    | 4.7   

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:

• M2 MacBook Air (2022): Runs at 150 RTFx real-time
• Performance scales with Apple Neural Engine capabilities

_{🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 35.1s diarization time • Test runtime: 2m 9s • 06/28/2026, 05:17 PM EST}